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Preface  &  Acknowledgements 


Welcome  to  our  Ninth  Annual  Acquisition  Research  Symposium!  This  event  is  the 
highlight  of  the  year  for  the  Acquisition  Research  Program  (ARP)  here  at  the  Naval 
Postgraduate  School  (NPS)  because  it  showcases  the  findings  of  recently  completed 
research  projects — and  that  research  activity  has  been  prolific!  Since  the  ARP’s  founding  in 
2003,  over  800  original  research  reports  have  been  added  to  the  acquisition  body  of 
knowledge.  We  continue  to  add  to  that  library,  located  online  at 

www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  60  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  hope  this  symposium  will  spark  even  more  participation. 

We  encourage  you  to  be  active  participants  at  the  symposium.  Indeed,  active 
participation  has  been  the  hallmark  of  previous  symposia.  We  purposely  limit  attendance  to 
350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 

We  expect  affordability  to  be  a  major  focus  at  this  year’s  event.  It  is  a  central  tenet  of 
the  DoD’s  Better  Buying  Power  initiatives,  and  budget  projections  indicate  it  will  continue  to 
be  important  as  the  nation  works  its  way  out  of  the  recession.  This  suggests  that  research 
with  a  focus  on  affordability  will  be  of  great  interest  to  the  DoD  leadership  in  the  year  to 
come.  Whether  you’re  a  practitioner  or  scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 

•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  &  Logistics) 

•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SFIIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 

•  Office  of  the  Assistant  Secretary  of  the  Air  Force  (Acquisition) 
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•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  & 
Technology) 

•  Deputy  Director,  Acquisition  Career  Management,  U.S.  Army 

•  Office  of  Procurement  and  Assistance  Management  Headquarters,  Department 
of  Energy 

•  Director,  Defense  Security  Cooperation  Agency 

•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Development,  Test  & 
Evaluation 

•  Program  Executive  Officer,  Tactical  Aircraft 

•  Director,  Office  of  Small  Business  Programs,  Department  of  the  Navy 

•  Director,  Office  of  Acquisition  Resources  and  Analysis  (ARA) 

•  Deputy  Assistant  Secretary  of  the  Navy,  Acquisition  &  Procurement 

•  Director  of  Open  Architecture,  DASN  (RDT&E) 

•  Program  Executive  Officer,  Littoral  Combat  Ships 

We  also  thank  the  Naval  Postgraduate  School  Foundation  and  acknowledge  its 
generous  contributions  in  support  of  this  symposium. 
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Fulbright  Scholar.  Dr.  Amara  worked  with  Shell  Oil  before  joining  the  Naval  Postgraduate  School.  She 
currently  researches  and  publishes  on  international  economics,  defense  economics,  health 
economics,  and  the  economics  of  the  public  sector.  She  has  addressed  various  national  and 
international  academic  organizations,  institutions  and  conferences.  Dr.  Amara  is  the  author  of  the 
forthcoming  book  Economic  Development  and  Post  Conflict  Reconstruction  and  co-editor  of  Military 
Medicine:  From  Pre-Deployment  to  Post-Separation.  She  has  published  in  numerous  peer-reviewed 
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research  and  development  in  the  area  of  modeling  and  simulation  supporting  the  U.S.  government 
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Abstract 

The  current  state  of  practice  in  program  risk  assessment  relies  on  the  best  judgment  of 
program  management  and  systems  engineering  to  identify  and  qualitatively  assess  the 
nature  and  probability  of  risks.  While  there  are  best  practices  and  lessons  learned  upon 
which  the  risk  assessment  process  can  rely,  the  process  is  still  heavily  qualitative  and  is 
performed  by  program  management  and  systems  engineering  staff  that  are  inherently 
optimistic  about  program  success.  As  a  result,  risks  and  their  consequences  are  frequently 
underestimated.  We  propose  a  methodology  that  actively  collects,  and  continuously  and 
quantitatively  analyzes,  metrics  that  are  earlier  indicators  of  risk  than  cost  and  schedule  slips. 
This  methodology  includes 

•  the  application  of  web-based  technologies  for  collection  and  analysis, 

•  a  quantified  risk  cloud  and  monetized  risk  thresholds, 

•  establishing  a  readily  accessible  knowledge  base  of  previous  program  failures,  and 

•  new  metrics  to  be  collected  closer  to  the  source  of  risk. 

Background 

The  Government  Accountability  Office  (GAO)  estimated  that  FY2008  acquisition 
costs  for  major  defense  programs  grew  by  $296  billion,  an  increase  of  25%  over  initial 
estimates  (GAO,  2009).  The  201 1  version  of  this  same  report  (GAO,  201 1 )  found  that 

the  total  acquisition  cost  of  the  programs  in  DOD’s  2010  portfolio  has 
increased  by  $135  billion  over  the  past  2  years,  of  which  $70  billion  cannot  be 
attributed  to  quantity  changes.  A  small  number  of  programs  are  driving  most 
of  this  cost  growth;  however,  half  of  DOD’s  major  defense  acquisition 
programs  do  not  meet  cost  performance  goals  agreed  to  by  DOD,  the  Office 
of  Management  and  Budget,  and  GAO. 

The  goal  of  this  paper  is  to  provide  a  better  approach  to  early  detection  of  program 
failure,  allowing  early  program  termination  or  course  correction.  Based  on  our  experience  on 
multiple  programs  of  various  sizes,  we  conclude  that,  despite  programmatic  assertions  to 
the  contrary,  risk  management  is  often  static  and  superficial.  It  is  one  of  the  areas  of 
program  management  that  is  not  well  supported  by  tools  that  produce  quantitative, 
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repeatable  results.  And  such  tools  are  only  possible  if  they  are  supported  by  a  knowledge 
base  of  previous  program  failures  and  lessons  learned.  Our  thesis  is  that  technology  and 
collaborative  techniques  have  evolved  to  the  point  that  a  quantitative,  data-driven  approach 
is  now  more  feasible  than  it  has  been  in  the  past.  This  data  should  come  from  two  sources: 

1 .  data  captured  from  previous  program  failures,  and 

2.  crowd-sourced  data  from  program  performers. 

We  believe  that  such  a  knowledge  base  and  tool  would  improve  the  process  of 
understanding  and  evaluating  risk  within  acquisition  programs  to 

•  reduce  program  failures,  cost  and  schedule  overruns,  and  unanticipated 
technical  and  managerial  roadblocks; 

•  better  anticipate  the  full  program  undertaking; 

•  prevent  repeating  historical  lessons  learned;  and 

•  provide  a  more  accurate  risk  analysis  to  existing  programs  to  have  a  clearer 
understanding  of  its  areas  of  predictability  and  unpredictability. 

Current  Risk  Management 

Figure  1  and  Table  1  illustrate  a  traditional  Continuous  Risk  Management  (CRM) 
process  for  acquisition  (NASA,  2000). 1  Unfortunately,  this  process  is  often  more  theoretical 
than  actual.  We  believe  our  proposed  approach  will  make  it  more  feasible  to  realize  this 
process. 


Figure  1.  Continuous  Risk  Management  Process 


1  The  Risk  Management  Guide  for  DoD  Acquisition  (2006)  advocates  a  process  that  is  only  slightly  different, 
placing  risk  mitigation  plan  implementation  (control)  before  tracking. 
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Table  1 .  Continuous  Risk  Management  Activities 


Activity 

Description 

Identify 

State  the  risk  in  terms  of  condition  and  consequences;  capture  the 
context  of  the  risk  (e.g.,  what,  when,  where,  how,  and  why). 

Analyze 

Evaluate  risk  probability,  impact/severity,  and  time-frame  (when  action 
needs  to  be  taken);  classify/group  with  similar/related  risks;  and  prioritize. 

Plan 

Assign  responsibility,  determine  approach  (research,  accept,  mitigate,  or 
monitor);  if  risk  will  be  mitigated,  define  mitigation  level  (e.g.,  action  item 
list  or  more  detailed  task  plan)  and  goal;  execute  plan. 

Track 

Acquire/update,  compile,  analyze,  and  organize  risk  data;  report  tracking 
results;  and  verify  and  validate  mitigation  actions. 

Control 

Analyze  tracking  results,  decide  how  to  proceed  (re-plan,  close  the  risk, 
invoke  contingency  plans,  continue  tracking);  execute  the  control 
decisions. 

Communication 

and 

Documentation 

These  are  present  in  all  of  the  preceding  functions  and  are  essential  for 
the  management  of  risks.  A  system  for  documentation  and  tracking  of 
risk  decisions  shall  be  implemented. 

Current  Tools 

Many  of  the  commercial  risk  analysis/assessment  tools  available  are  focused  on  an 
enterprise’s  information  technology  (IT;  including  information  security)  and  facilities,  for 
example,  Central  Computing  and  Telecommunications  Agency  (CCTA,  n.d.)  Risk  Analysis 
and  Management  Method  (CRAMM),  Cost  of  Risk  Analysis  (CORA,  n.d.),  and 
Countermeasures  (n.d.).  This  is  not  surprising  since  a  commercial  tool  must  demonstrate 
immediate  value,  which  argues  for  focusing  on  risk  domains  where  large  data  sets  of 
discrete,  identifiable  risks  exist. 

The  Defense  Acquisition  Portal  (n.d.)  lists  several  risk  assessment  tools: 

•  ACEIT  (Automated  Cost  Estimating  Integrated  Tools)  RI$K  is  one  of  a  family  of 
applications  that  support  project  managers  and  cost/financial  analysts  during  all 
phases  of  a  program’s  life  cycle  that  permits  the  user  to  conduct  detailed  cost- 
risk  analysis.  It  provides  risk-adjusted,  phased  cost  estimates,  and  performs  risk 
calculations  using  eight  probability  functions  to  model  uncertainty  (beta, 
lognormal,  log-f,  normal,  student’s  t,  triangular,  uniform,  and  Weibull). 

•  Army  Cost  Analysis  Strategy  Assessment  (CASA)  is  a  Life  Cycle  Cost 
(LCC)/Total  Ownership  Cost  (TOC)  decision  support  tool  that  includes  risk  and 
uncertainty  analysis  as  a  capability. 

•  Cost/Risk  Identification  &  Management  System  (CRIMS)  is  a  system  for 
identifying,  tracking,  and  storing  cost-risk  information.2 

•  Formal  Risk  Analysis  (FRISK  4.00)  supports  cost-risk  analysis  by  allowing  the 
user  to  statistically  sum  Work  Breakdown  Structure  (WBS)-element  costs, 
represented  by  probability  distributions,  to  obtain  a  probability  distribution  of  total 
cost.  This  appears  to  indicate  monetization  of  risk,  but  the  most  current  version  is 
from  2000. 


2  Current  information  on  this  tool  was  unavailable. 
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•  NetIDEAS  Inc. — Risk  Manager  supports  the  suggestions  made  in  this  paper  in 
terms  of  using  an  online  environment  for  team  members  to  collaborate  on  risk 
identification,  assessment,  mitigation,  and  contingency  planning. 

•  Technical  Risk  Identification  and  Mitigation  System  (TRIMS)  purports  to  have  a 
knowledge  base.  However,  information  on  its  source  was  unavailable,  and  it 
does  not  appear  to  allow  for  collaboration  outside  the  program  using  it. 

Our  key  observations  about  these  tools  are  as  follows: 

1 .  They  are  focused  mostly  on  methodology  rather  than  data. 

2.  Many  are  not  maintained  actively,  suggesting  that  they  have  established  an 
initial  methodology  and  have  no  impetus  to  evolve  past  that  methodology.  By 
extension,  the  tool  that  implements  it  does  not  evolve  because  the  tool  is  not 
integrated  with  a  risk  management  enterprise  that  tracks  project  results. 

Shortcomings  of  Current  Practice 

In  practice,  we  have  observed  the  following  shortcomings  of  risk  assessments  on 
actual  programs: 

•  Senior  acquisition  team  members,  especially  management,  have  a  vested 
interest  in  underestimating  risk.  This  is  not  only  because  of  their  desire  for 
continuity  of  the  program,  but  because  their  role  requires  an  optimistic  attitude  to 
lead  their  program  team  to  success. 

•  Engineers  often  recognize  technical  and  programmatic  risks,  but  cannot/will  not 
risk  raising  concern  for  fear  of  retaliation  or  working  at  cross-purposes  with  their 
leaders’  success-oriented  optimism. 

•  Engineers  are  rarely  included  in  the  risk  assessment  process. 

•  All  acquisition  team  members  are  limited  by  their  collective  knowledge  of  risk.  No 
matter  how  experienced  the  team  is,  team  members  cannot  be  expected  to  know 
or  recognize  all  the  risks  that  have  ever  resulted  in  acquisition  failures  or  their 
indicators. 

•  Risks  are  usually  assessed  qualitatively  rather  than  quantitatively  because  team 
members  lack  the  data  to  produce  useful  and  realistic  metrics. 

•  Methodologies  that  look  at  cost  and  schedule  (e.g.,  Earned  Value  Management 
[EVM])  are  assessing  symptoms,  not  causes. 

•  Well-defined  metrics  are  needed  that  are  causally  closer  to  symptoms  or 
program  risk. 

•  The  potential  causal  relationships  between  individual  risks  are  often  overlooked 
(i.e.,  a  failure  in  one  aspect  of  a  program  may  have  a  cascade  effect). 

Proposed  Risk  Analysis  Snapshot  for  Acquisition  Program  Cost  Projection 

A  well-managed  program  can  sustain  some  number  of  unexpected  events  and  stay 
within  budget.  The  question  to  be  answered  is,  how  many  unexpected  events,  and  of  what 
magnitude,  can  be  tolerated  before  program  recovery  is  not  achievable?  By  monetizing 
program  risks,  a  probabilistic  risk  “cloud”  can  be  calculated  that  permits  understanding  of  the 
additional  costs  that  unexpected  events  will  incur  for  the  program.  Figure  2  provides  a 
conceptual  illustration  of  this  risk  cloud.  When  the  risk  cloud  is  added  to  the  top  of  the 
expected  costs,  a  clearer  picture  of  the  possible  overall  costs  that  a  program  may  incur  are 
seen.  The  challenge  with  this  simple  picture  is  that  early  in  a  project,  when  the  current 
expended  funds  are  low,  the  project  budget  has  quite  a  bit  of  capability  to  absorb  future 
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project  changes  due  to  unexpected  decisions  and  project  direction.  The  risk  cloud  above  the 
project  budget  should  be  small  at  that  time.  As  the  project  matures,  and  more  funds  have 
been  expended,  the  size  of  the  probabilistic  risk  cloud  changes. 


—  Maximum  allowable  expenditure  (defined  by  customer) 

—  Probabilistic  risk  cloud  representing  cost  of  unexpected  events 

—  Project  budget 


—  Current  expended  funds 


Figure  2.  Risk  Cloud 

By  conceptualizing  the  cost  of  a  program  in  this  manner,  the  upper  financial  limit  that 
would  be  allowed,  even  through  an  overrun,  can  be  defined.  That  limit  can  be  called  the 
Maximum  Allowable  Expenditure  (MAE).  If  an  overrun  were  borne  by  the  customer,  then  the 
customer  would  define  the  MAE.  In  that  case,  the  customer  may  justifiably  terminate  or 
significantly  reorganize  a  program  if  the  estimated  average  of  unexpected  events  will  drive 
costs  above  the  MAE.  If  the  contract  were  firm  fixed  price,  then  the  MAE  would  be  defined 
by  the  contractor,  who  would  bear  the  cost  of  an  overrun.  In  both  cases,  the  customer, 
program  manager,  and  contractor  are  very  concerned  with  a  risk  cloud  that  approaches  or 
surpasses  the  MAE. 

The  project  manager  might  also  set  the  MAE  low  enough  to  prevent  loss  of 
incentives  such  as  award  fees.  However,  cost  elasticity  is  not  out  of  the  question  as  the 
enactment  of  Nunn-McCurdy  (§  2433)  so  strongly  demonstrates.  One  of  the  primary  reasons 
to  implement  the  recommendations  in  this  paper  is  to  avoid  the  type  of  visibility  associated 
with  program  cost  overruns  and  a  required  explanation  to  Congress. 

The  MAE  should  vary  over  the  lifetime  of  a  program.  Although  uncertainty  may  be 
high,  the  risk  of  program  failure  should  be  low  at  the  beginning  of  a  program  since  there  is 
greater  time  and  budget  to  recover.  Figure  3  illustrates  the  concept  and  projected  evolution 
of  the  average  and  maximum  of  the  risk  cloud  over  the  course  of  a  program. 
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Figure  3.  Projected  Risk  Evolution  Over  Time 


ACQUISITION  RESEARCH  PROGRAM: 
CREATING  SYNERGY  FOR  INFORMED  CHANGE 


-56- 


Notice  that  the  MAE  is  rising  until  the  “middle”  or  risk  peak  of  the  program  and  falling 
toward  the  end.  Figure  3  clearly  reflects  perfect  knowledge  of  how  risk  will  evolve  on  the 
program,  the  type  of  knowledge  a  project  manager  (PM)  almost  never  has  in  reality.  The  two 
key  points  about  this  profile  are  the  following: 

•  The  change  over  time  reflects  the  fact  that  the  PM  is  anticipating  there  is  a  knee 
in  the  risk  curve  while  keeping  a  close  rein  on  the  risk. 

•  Any  time  the  top  edge  of  the  risk  cloud  crosses  or  is  about  to  cross  the  MAE,  it 
should  trigger  a  Program  Scrub  Decision  (PSD)  to  cancel  or  reorganize  the 
program. 

For  this  approach  to  be  effective,  the  MAE  profile  needs  to  anticipate  rising  and 
falling  risk.  If  the  MAE  were  flat  across  the  program  timeline,  and  it  were  set  too  high  initially, 
the  program  could  get  into  trouble  early  without  triggering  a  PSD. 

Figure  4  illustrates  what  could  happen  if  the  MAE  were  flat,  but  set  too  low  initially. 

As  the  risk  rises  early  in  the  program,  a  PSD  is  triggered.  If  the  PM  concludes  that  the  risk  is 
still  manageable,  the  MAE  is  raised.  But  if  the  MAE  profile  is  still  flat,  risk  could  continue  to 
rise  toward  the  end  of  the  program  rather  than  falling  as  expected  without  triggering  another 
PSD. 
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Figure  4.  Program  Scrub  Decision  Point 


A  concept  that  needs  to  be  better  understood  is  where  the  knee  of  the  curve  exists 
(as  shown  in  Figure  3;  i.e.,  where  to  allow  the  greatest  risk).  This  may  be  related  to  the 
Technical  Readiness  Level  (TRL;  DoD,  2011)  of  the  program. 


Recommendations  for  a  “Living”  Risk  Management  Capability 

The  quantitative  process  described  in  the  following  subsection  is  only  feasible  with 
the  support  of  data  and  the  infrastructure  to  make  it  usable  and  accessible  by  programs. 


Quantitative  Process 

The  quantitative  process  helps  determine  the  type  of  data  that  need  to  be  gathered. 
The  process  is  as  follows: 


•  Identify  an  initial  set  of  discrete  risk  elements. 

•  Monetize  the  identified  set  of  discrete  risk  elements  based  on  their  negative 
impact  to  project  resources  (duration/schedule,  personnel,  and  materials) 
multiplied  against  the  likelihood  of  occurrence  (based  on  past  acquisition 
project  histories). 
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•  Determine  the  mathematics  of  unifying  the  monetized  risk  across  the  project 
by  determining  the  causal  relationships  between  the  discrete  risk  elements, 
based  on  the  Eight-Stage  Risk  Assessment  Methodology  (Drake  &  Morse, 
1994): 

o  If  there  is  no  causal  relationship,  then  a  simple  summation  across  the 
monetized  risk  elements  is  sufficient. 

o  If  risk  elements  are  mutually  exclusive,  then  selecting  the  maximum  of 
the  risk  is  sufficient. 

o  If  risk  elements  are  causally  chained  together,  then  the  Eight-Stage 
Risk  Assessment  Methodology  (or  a  similar  calculation)  applies, 
where  the  causal  elements  will  raise  the  monetized  value  due  to  the 
likelihood  of  the  combined  probability  of  occurrence. 

•  Continually  track,  update,  and  plan  against  risks. 

•  Compare  their  risks  to  a  substantial  knowledge  base  of  risks  from  previous 
programs. 

•  Collect  metrics  to  support  quantitative  risk  adjustments. 

Data 

The  portal  should  be  backed  by  a  knowledge  base  of  previous  acquisition  successes 
and  failures  detailing 

•  types  of  failures; 

•  indicators  of  failures; 

•  probability  of  failure  based  on  indicators; 

•  loss  percentages  based  on  failures;  and 

•  applicable  risk  treatments  (remedies),  including  success  and  failure  metrics. 

Infrastructure 

The  web  portal  that  supports  the  process  in  the  Quantitative  Process  section  will 
need  the  following  functionality: 

•  knowledge  base  of  individual  risks  searchable  on  multiple  criteria  (e.g.,  phase 
technical/programmatic,  technologies,  drivers); 

•  management  tools  for  continually  tracking,  updating,  and  planning  against  risks; 

•  metrics  collection  tools;  and 

•  open  APIs  for  importing  and  exporting  data. 

Usability 

The  portal  should  have  the  following  features: 

•  Present  potential  risks  from  which  the  risk  assessor  can  choose,  reducing  the 
effort  to  “think  up”  all  potential  risks,  but  still  allow  the  assessor  to  specify  new 
risks. 

•  Provide  recommendations  and  guidance  on  techniques  that  apply  to 
assessing  individual  risks:  for  example, 

o  Failure  Modes  and  Effects  Analysis  (FMEA), 

o  Fault  Tree  Analysis  (FTA), 
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o  Probabilistic  Risk  Assessment  (PRA),  and 
o  Probability  distributions  appropriate  for  individual  risks. 

•  Integrate  tools  for  collecting  metrics,  including  anonymous  polling  of  team 
members.3 

The  portal  should  provide  the  following  key  outputs: 

•  Cloud  of  probable  cost  based  on  risk  and  comparison  of  the  risk  cloud  to 

o  target  cost, 
o  actual  cost, 
o  MAE, 

o  average  risk,  and 
o  monetized  risk. 

•  Tracking  and  adjustment  of  risk  probabilities  based  on  previous  program 
performance. 

o  If  the  program  has  consistently  underestimated  risks,  then  future  risk 
probabilities  should  be  increased  accordingly. 

•  Versioning  and  tracking  to  enable  rapid  assessment  of  risk  management 
success  over  time. 


Accessibility 

The  portal  should  have  role-based  access  control  for 

•  management, 

•  engineers, 

•  government  program  managers, 

•  auditors,  and 

•  portal  maintenance  staff. 

Deploying  and  Employing  the  Portal 

While  the  application  of  this  portal  is  technically  outside  the  scope  of  our  effort  to 
improve  the  process  of  monetizing  risk,  the  broad  adoption  of  the  proposed  process  and 
portal  could  factor  into  contracts  in  the  form  of  specified  government  remedies  (e.g., 
cancellation  or  re-bid)  tied  to  specific  risk  metrics. 

Some  Proposed  (and  Potentially  Unpopular)  Risk  Metrics 

At  the  end  of  the  subsection  Current  Risk  Management,  we  asserted  the  need  for 
metrics  that  are  causally  closer  to  symptoms  rather  than  those  that  capture  and  record  their 
impact  later,  such  as  EVM.  Table  2  presents  some  proposed  metrics  from  our  personal 
experiences  on  programs  of  various  sizes. 


3  Engineers  are  often  aware  of  technical  risks  before  PMs,  since  they  are  “closer”  to  the  fault,  and  it  is  usually  the 
engineers  that  have  to  relay  the  technical  issue  to  the  manager. 
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Table  2.  Proposed  New  Risk  Metrics 

Metric 

Description 

Work-to-noise 
ratio  =  work/(work 
+  noise) 

•  What  percentage  of  time  do  engineers  spend  actually  performing 
engineering  tasks  vs.  engaging  in  non-productive  tasks  (e.g.,  sitting 
in  meetings/telecons  to  which  they  are  not  contributing  nor  from 
which  they  are  getting  actionable  information)? 

•  This  metric  can  be  collected  anonymously  through  the  proposed 
web  portal. 

Task  coupling 

•  To  what  degree  does  completion  of  tasks  depend  on  coordination 
between  tasks? 

•  This  metric  is  related  to  a  Gantt  chart  of  the  schedule,  but  cannot 
be  seen  on  the  Gantt  chart. 

•  Task  coupling  not  only  runs  the  risk  of  making  both  tasks  late 
because  they  are  interdependent,  but  simply  trying  to  arrange  the 
coordination  often  delays  the  tasks  because  schedules  are  hard  to 
synchronize. 

Resource  gapping 

•  A  delay  in  one  task  delays  a  dependent  task,  causing  a  gap  in 
tasking  for  some  resources. 

o  This  could  be  observed  as  resource  underutilization  in  the 
schedule. 

•  Unlike  machinery,  engineers  cannot  just  sit  idle;  they  have  to  be 
retained  and  paid,  so  they  have  to  perform  work. 

o  It  must  be  value-adding  or  the  costs  are  wasted. 

o  They  cannot  be  shifted  to  other,  productive  tasks  because 
other  resources  have  already  been  allocated,  and  putting  them 
on  other  programs  risks  losing  them  and  their  expertise 
altogether. 

Engineering  skill 
loss 

•  This  includes  not  just  the  loss  of  engineers,  but  the  loss  of  the  most 
knowledgeable  and  skilled  engineers. 

•  Engineers  are  not  interchangeable  resources. 

o  When  rumors  of  cuts  start,  the  better  engineers  find  new  jobs 
and  leave. 

o  The  loss  of  engineering  skill  is  disproportionally  larger  than  the 
loss  of  engineers. 

Energy  drains 

•  These  are  individuals  who  have  psychological  and  substantive 
negative  impacts  on  otherwise  productive  members  of  the  team, 
o  Leaders  who  demoralize  subordinates  to  the  point  that  they 
waste  time  complaining  to  each  other  rather  than  working 
o  Team  members  who  perform  “negative  work”  (i.e.,  whose  work 
is  so  poor  that  it  takes  more  time  to  correct  than  it  would  have 
taken  a  productive  team  member  to  do  it  in  the  first  place). 

These  individuals  also  negatively  affect  morale  because 
productive  team  members  resent  having  to  work  harder  to 
make  up  for  the  counter-productivity. 

Next  Steps  and  Challenges 

We  cannot  switch  from  qualitative  analysis  to  quantitative  analysis  without 
quantitative  data.  Central  to  the  success  of  this  approach  is  the  collection  and  maintenance 
of  data  on  previous  acquisition  risks  and  failures.  While  some  of  this  data  may  be  available, 
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it  will  need  to  be  evaluated  to  determine  if  it  is  sufficiently  detailed  and  quantified  to  be 
transformed  into  failure  records  as  described  in  the  Data  subsection.  There  are  a  number  of 
potential  sources  coordinated  by  USD(AT&L).  The  data  for  major  programs  are  collected  in 
the  Defense  Acquisition  Executive  Summary  (DAES,  n.d.),  a  multi-part  document  reporting 
program  information  and  assessments  that  provides  an  early-warning  report  to  USD(AT&L). 
The  DAES  describes  actual  program  problems,  warns  of  potential  program  problems,  and 
describes  mitigating  actions  taken  or  planned.  Another  potential  source  is  AT&L’s 
Performance  Assessment  and  Root  Cause  Analyses  (PARCA)  office  (PARCA,  n.d.),  the 
central  office  for  major  defense  authorization  performance  assessment,  root  cause  analysis, 
and  EVM. 

The  success  of  the  proposed  portal  is  reliant  upon  continuous  acquisition  of  new 
data  that  is  a  key  challenge  of  any  such  enterprise.  Program  personnel  must  be  incentivized 
to  share  experiences  and  may  require  anonymity  to  do  so.  New  metrics  should  be  solicited 
from  the  community.  Acceptable  values  for  some  of  the  new  metrics  and  their  impact  on  risk 
(e.g.,  work-to-noise  ratio)  cannot  be  determined  without  data  collection  from  multiple 
programs. 

Finally,  this  type  of  community  effort  succeeds  and  thrives  when  the  community 
continuously  rates  the  value  of  the  elements  of  the  enterprise.  In  this  case,  the  applicability 
of  identified  risks  and  the  accuracy  of  their  associated  metrics  should  be  continuously 
evaluated  and  adjusted  based  on  community  feedback,  improving  accuracy  over  time. 
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Summary  Proposal 


We  propose  a  methodology  that  actively  collects  and  continuously, 
quantitatively  analyzes  metrics  that  are  earlier  indicators  of  risk  than  cost 
and  schedule  slip.  This  methodology  includes: 

■The  application  of  web-based  technologies  to  collection  and  analysis 

■A  quantified  risk  cloud  and  monetized  risk  thresholds 

"Establishing  a  readily-accessible  knowledge  base  of  previous  program 
failures 

"New  metrics  to  be  collected  closer  to  the  source  of  risk 


dPL 


2 


Traditional  Continuous  Risk  Management  (CRM) 
Process  for  Acquisition 


•  Identify:  State  the  risk  in  terms  of  condition  and 

consequences;  capture  the  context  of  the  risk;  e.g.,  what, 
when,  where,  how,  and  why. 


•  Analyze:  Evaluate  risk  probability,  impact/severity,  and 
time-frame  (when  action  needs  to  be  taken);  classify/group 
with  similar/related  risks;  and  prioritize. 


•  Plan:  Assign  responsibility,  determine  approach  (research, 
accept,  mitigate,  or  monitor);  if  risk  will  be  mitigated,  define 
mitigation  level  (e.g.,  action  item  list  or  more  detailed  task 
plan)  and  goal;  execute  plan. 


•  Track:  Acquire/update,  compile,  analyze,  and  organize  risk 
data;  report  tracking  results;  and  verify  and  validate 
mitigation  actions. 


•  Control:  Analyze  tracking  results,  decide  how  to  proceed 
(re-plan,  close  the  risk,  invoke  contingency  plans,  continue 
tracking);  execute  the  control  decisions. 


•  Communication  and  documentation:  These  are  present 
in  all  of  the  preceding  functions  and  are  essential  for  the 
management  of  risks.  A  system  for  documentation  and 
tracking  of  risk  decisions  shall  be  implemented. 
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Observations  About  Existing  Risk  Assessment 
Approaches 


■  Team  members  tend  toward  optimism,  consequently  underestimating  risk. 

■  Engineers  often  recognize  technical  and  programmatic  risks,  but 
can’t/won’t  risk  raising  concern  for  fear  of  retaliation. 

•  Engineers  are  rarely  included  in  risk  assessment. 

■  All  acquisition  team  members  are  limited  by  their  collective  knowledge  of 
risk. 

•  They  can’t  be  expected  to  know  or  recognize  all  the  risks  that  have  ever 
resulted  in  acquisition  failures. 

■  Risks  are  usually  assessed  qualitatively  rather  than  quantitatively  because 
they  lack  the  data  to  produce  useful  and  realistic  metrics. 

-  Methodologies  that  look  at  cost  and  schedule,  e.g.  EVMS,  are  assessing 
symptoms,  not  causes. 

•  We  need  metrics  that  are  causally  closer  to  symptoms. 

■  The  potential  causal  relationships  between  individual  risks  are  often 
overlooked,  i.e.  a  failure  in  one  aspect  of  a  program  may  have  a  cascade 
effect. 
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Cost 


Proposed  Risk  Analysis  Snapshot  for  Acquisition 
Project  Cost  Projection 


■  By  monetizing  project  risks,  a  probabilistic  risk  “cloud”  can  be  calculated  that 
permits  understanding  of  the  additional  costs  unexpected  events  will  incur  for 
the  project. 

■  A  well-managed  project  can  sustain  a  set  of  unexpected  events  and  stay  within 
budget. 

■  A  sponsor  may  justifiably  terminate  (or  significantly  reorganize)  a  project  if  an 
average  of  unexpected  events  will  drive  costs  above  the  maximum  allowable 
expenditure. 

•  The  performer  might  also  set  the  maximum  expenditure  to  prevent  loss  of 
a  incentives  such  as  award  fees. 

Maximum  allowable  expenditure  (defined  by  customer) 

Probabilistic  risk  cloud  representing  cost  of  unexpected  events 
Project  budget 


Current  expended  funds 
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Cost 


Proposed  Risk  Analysis  for  Acquisition  Project  Cost 
Projection  Over  the  Acquisition  Lifecycle 


■  Risk  level  should  be  low  at  the  beginning  of  the  project  since  there  is 
greater  room  to  recover. 

-  A  concept  that  needs  to  be  better  understood  is  where  the  knee  of  the 
curve  exists,  i.e.  where  to  allow  the  greatest  risk. 

•  This  may  be  related  to  the  Technical  Readiness  Level  (TRL)  of  the 
project. 
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Cost 


Project  Scrub  Decision  Point 


TSKJ 


■  At  any  time,  if  the  risk  cloud  is  projected  to  cross  the  maximum  allowed 
expenditure,  there  is  a  critical  decision  point: 

•  Raise  the  maximum  allowed  expenditure  or 

•  Cancel  or  reorganize  the  program 
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Recommendations  for  a  “Living”  Risk  Management 
Capability 


"  A  quantitative  process  as  previously  described  is  only  feasible  with  the 
support  of  data  and  the  infrastructure  to  make  it  usable  and  accessible 
by  programs. 
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Quantitative  Process 


■  Identify  an  initial  set  of  discrete  risk  elements 

■  Monetize  the  identified  set  of  discrete  risk  elements  based  on  their 
negative  impact  to  project  resources  multiplied  against  the  likelihood  of 
occurrence 

■  Determine  the  mathematics  of  unifying  the  monetized  risk  across  the 
project  by  determining  the  causal  relationships  between  the  discrete 
risk  elements 

■  Continually  track,  update,  and  plan  against  risks 

■  Compare  their  risks  to  a  substantial  knowledge  base  of  risks  from 
previous  programs 

■  Collect  metrics  to  support  quantitative  risk  adjustments 
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Data 


■  Types  of  failures 

■  Indicators  of  failures 

■  Probability  of  failure  based  on  indicators 

■  Loss  percentages  based  on  failures 

■  Applicable  risk  treatments  (remedies)  including  success  and  failure 
metrics 
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Infrastructure 


■  Knowledge  base  of  individual  risks  searchable  on  multiple  criteria,  e.g.  phase 
technical/programmatic,  technologies,  drivers 

■  Management  tools  for  continually  tracking,  updating,  and  planning  against  risks 

■  Metrics  collection  tools 

■  Open  APIs  for  importing  and  exporting  data 
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Usability 


■  Portal  features: 

•  Present  potential  risks  from  which  the  risk  assessor  can  choose, 
reducing  the  effort  to  “think  up”  all  potential  risks,  but  still  allow  the 
assessor  to  specify  new  risks. 

•  Provide  recommendations  and  guidance  on  techniques  that  apply  to 
assessing  individual  risks 

•  Integrate  tools  for  collecting  metrics  including  anonymous  polling  of 
team  members 

■  Outputs: 

•  Cloud  of  probable  cost  based  on  risk  and  comparison  of  the  risk 
cloud 

•  Tracking  and  adjustment  of  risk  probabilities  based  on  previous 
program  performance. 

•  Versioning  and  tracking  to  enable  rapid  assessment  of  risk 
management  success  over  time 
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Accessibility 


■  Role-based  access  control  for: 

•  Management 

•  Engineers 

•  Government  program  managers 

•  Auditors 

•  Portal  maintenance  staff 
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Deploying  and  Employing  the  Portal 


TSKJ 


■  While  the  application  of  this  tool  is  technically  outside  the  scope  of  our 
effort  to  improve  the  process  of  monetizing  risk,  the  broad  adoption  of 
the  proposed  process  and  tool  could  factor  into  contracts  in  the  form  of 
specified  government  remedies,  e.g.  cancellation  or  re-bid,  tied  to 
specific  risk  metrics. 
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Some  Proposed  (and  Potentially  Unpopular)  Risk 
Metrics  (1  of  2) 


■  Work  to  noise  ratio  =  work  /  (work  +  noise) 

•  What  percentage  of  time  do  engineers  spend  actually  performing 
engineering  tasks  vs  engaging  in  non-productive  tasks,  e.g.  sitting  in 
meetings/telecons  to  which  they’re  not  contributing  nor  from  which 
they’re  getting  actionable  information? 

•  This  metric  can  be  collected  anonymously  through  the  proposed  web 
portal. 

■  Task  coupling 

•  To  what  degree  does  completion  of  tasks  depend  on  coordination 
between  tasks? 

•  This  metric  is  related  to  a  Gantt  chart  of  the  schedule,  but  can’t  be 
seen  on  the  Gantt  chart. 

•  Task  coupling  not  only  runs  the  risk  of  making  both  tasks  late 
because  they’re  interdependent,  but  simply  trying  to  arrange  the 
coordination  often  delays  the  tasks  because  schedules  are  hard  to 
synchronize. 

- dPL 
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Some  Proposed  (and  Potentially  Unpopular)  Risk 
Metrics  (2  of  2) 


■  Resource  gapping 

•  A  delay  in  one  task  delays  a  dependent  task,  causing  a  gap  in  tasking  for  some  resources. 

>  This  could  be  observed  as  resource  underutilization  in  the  schedule. 

•  Unlike  machinery,  engineers  can’t  just  sit  idle;  they  have  to  be  retained  and  paid,  so  they 
have  to  perform  work. 

>  It  must  be  value-adding  or  the  costs  are  wasted. 

>  They  can’t  be  shifted  to  other,  productive  tasks  because  other  resources  have  already 
been  allocated,  and  putting  them  on  other  programs  risks  losing  them  and  their  expertise 
altogether. 

■  Engineering  skill  loss 

•  Not  just  the  loss  of  engineers,  but  the  loss  of  the  most  knowledgeable  and  skilled  engineers. 

•  Engineers  are  not  interchangeable  parts. 

>  When  rumors  of  cuts  start,  the  better  engineers  find  new  jobs  and  leave. 

>  The  loss  of  engineering  skill  is  disproportionally  larger  than  the  loss  of  engineers. 

■  Energy  drains 

•  Individuals  who  have  psychological  and  substantive  negative  impacts  on  otherwise 
productive  members  of  the  team 
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DATA! 


■  We  can’t  switch  from  qualitative  analysis  to  quantitative  analysis 
without  quantitative  data. 

■  Central  to  the  success  of  this  approach  is  the  collection  and 
maintenance  of  data  on  previous  acquisition  risks  and  failures. 

■  New  metrics  should  be  solicited  from  the  community. 

•  Acceptable  values  for  some  of  the  new  metrics  and  their  impact  on 
risk,  e.g.  work  to  noise  ratio,  cannot  be  determined  without  data 
collection  from  multiple  programs. 

■  This  type  of  community  effort  succeeds  and  thrives  when  the 
community  continuously  rates  the  value  of  the  elements  of  the 
enterprise. 

•  In  this  case,  the  applicability  of  identified  risks  and  the  accuracy  of 
their  associated  metrics  should  be  continuously  evaluated  and 
adjusted  based  on  community  feedback,  improving  accuracy  over 
time. 


dPL 


17 


References 


1.  U.S.  Government  Accountability  Office,  Assessments  of  Selected  Weapon  Programs,  GAO-09-326SP, 
March  30,  2009,  p.  Highlights. 

2.  U.S.  Government  Accountability  Office,  Assessments  of  Selected  Weapon  Programs,  GAO-1 1 -233SP, 
March  30,  2011,  p.  Highlights. 

3.  NASA,  Risk-Based  Acquisition  Management  (R-BAM),  July  27,  2000. 

4.  The  Risk  Management  Guide  for  DoD  Acquisition,  Sixth  Edition,  August  2006. 

5.  Central  Computing  and  Telecommunications  Agency  (CCTA)  Risk  Analysis  and  Management  Method 
(CRAMM),  http://www.cramm.com. 

6.  Cost  of  Risk  Analysis,  http://www.softscout.com/software/Project-and-Business-Manaqement/Risk- 
Manaqement/CORA--Cost-of-Risk-Analysis.html. 

7.  Countermeasures,  http://www.countermeasures.com. 

8.  https://dap.dau. mil/aphome/das/Lists/Software%20Tools/All.aspx?sgroup=AII 

9.  DoD  Authorization  Act,  1983  (P.L.  97-252),  Nunn-McCurdy  Act  (10  U.S.C.  §  2433) 

10.  Defense  Acquisition  Guidebook,  DoD  7/29/2011,  Chapter  10.5.2.2. 

11.  David  L.  Drake  and  Katherine  L.  Morse,  "The  Security-Specific  Eight  Stage  Risk  Assessment 
Methodology,"  Proceedings  of  the  17th  National  Computer  Security  Conference,  1994. 

12.  Defense  Acquisition  Executive  Summary,  https://acc.dau.mil/adl/en- 
US/24422/file/2852/Defense%20Acquisition%20Executive%20Summary%20(DAES).doc 

13.  Performance  Assessment  and  Root  Cause  Analyses,  http://www.acq.osd.mil/parca/about.shtml 


dPL 


18 


BACKUPS 


Graphics 


Objective 


■  Improve  the  process  of  understanding  and  evaluating  risk  within 

acquisition  projects  to: 

•  Reduce  project  failures,  cost  and  schedule  overruns,  and 
unanticipated  technical  and  managerial  roadblocks 

•  Better  anticipate  the  full  project  undertaking 

•  Prevent  repeating  historical  lessons  learned 

•  Provide  a  more  accurate  risk  analysis  to  existing  projects  to  have  a 
clearer  understanding  of  its  areas  of  predictability  and 
unpredictability 
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Task  Coupling 
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